Comments on articles in RESS special issue

The authors are to be congratulated on their attempt to extract results from the unique Delft database. Their conclusions are interesting and thought-provoking. However, I am concerned about the analysis of individual question effects. As in so many aspects of elicitation, I think it is helpful to disentangle the uncertainties due to lack of knowledge, i.e. epistemic uncertainty, from those due to intrinsic randomness, i.e. aleatory uncertainty. Consider the questions in the Dike Ring study that are listed in Table 2. All of these questions either implicitly or explicitly ask for opinions about individual instances which are subject to aleatory uncertainty. For instance, question Hs asks about wave height for “a randomly chosen occurrence” (of some phenomenon). The randomness here and in question Ts is explicit, but it is implicit in the other questions. Of course, the expert will also have epistemic uncertainty concerning properties of the population of random instances, such as the mean. In general, I believe it is good practice in elicitation to ask separately about sources of epistemic and aleatory uncertainty, and before continuing to discuss this paper I think it is worthwhile elaborating a little on my view because it is not that of those whose elicitations are found in the Delft database. Theirs is the view that we should only ever ask experts about observable random variables. According to that view, the mean ratio of wave heights (in the population of all occurrences of the phenomenon under study) is not observable and so experts’ opinions about it should not be elicited. In contrast, it is my experience that it is perfectly possible to conduct meaningful elicitation about such quantities, and that to do so has the important benefit of separately eliciting epistemic uncertainty. Amongst other advantages, this helps to counter over-confidence (which is major concern of the present paper), and by modelling dependencies appropriately it avoids having to elicit them. Returning to the Dike Ring study, the authors find better calibration on questions Ts and Hs than on the others, and I suggest that this may be at least in part due to the fact that the aleatory uncertainty is more explicit in the wording of these questions. Respondents to the other questions may have failed to acknowledge the randomness fully, concentrating instead on uncertainty about the mean. This in itself would explain the higher degree of over-confidence found for those questions.